Information terminal

ABSTRACT

An information terminal that can perform image processing in consonance with the use state and the use purpose. When a retrial module is started, removal means removes part or all of the interpolation processing performed for a Bayer-type module. Thereafter, data obtained by the removal are transmitted to color interpolation means, another color interpolation process is performed for the data, and the resultant data are transmitted to image quality correction means. The image quality correction means performs another image quality correction process for the data, and transmits the obtained data to JPEG encoding means.

TECHNICAL FIELD

The present invention relates to an information information terminal,and specifically to an information terminal including an interpolationdevice.

BACKGROUND ART

Recently, information terminals equipped with cameras have becomepopular, and have also been employed as means for imaging, browsing andimage communication.

For example, in the case of a cellular phone equipped with a camera, animage acquired by the camera is used in various ways, such as beingtransmitted as a file attached to email text, being used for real-timecommunication (a television telephone), or being printed by a user. Thesame uses may be made of an image received from a third party and of animage downloaded from a server.

Therefore, originally, in order to increase the usability, it ispreferable that the image quality or the speed required for theprocessing, i.e., the amount of processing for an operation or theamount of processing memory, be changed in accordance with the use of aphotographic image or a received image, as described above.

For example, as shown in FIG. 5, “processing speed has priority overimage quality” when a photographic image is to be used for real-timecommunication. On the other hand, for example, when a photographic imageis to be printed out, a case wherein “image quality takes priority overprocessing speed” is more meritorious for a user.

DISCLOSURE OF THE INVENTION

However, generally, a conventional cellular phone merely provides afixed image quality and a fixed processing speed, regardless of what isfavored by a user and its use.

Furthermore, since downsizing and a reduction in power consumption andits costs are indispensable for an information terminal, such as acellular phone, it is not easy to mount a camera, such as describedabove, that performs flexible image processing.

Further, since downsizing, a reduction in power consumption and costsare also indispensable for the processor of an information terminal, itis not generally projected that all of the image processing ordinarilyperformed by a camera be shifted to the processor in order to guaranteethe flexibility of the camera image processing.

While taking the above points into account, one objective of the presentinvention is to provide an information terminal that can minimizealterations of a camera and a processor of the information terminal, andthat can flexibly change image quality and processing speeds, at a lowcost and within a short period of time, in accordance with the usagestates and other, varied conditions, such as the extent of operations,required of the imaging processing performed by the camera and theinformation terminal, and the amount of processing memory required forthe operation.

It is another objective of the present invention to provide aninformation terminal that can reduce the influences due to thesimplified processing, performed by a camera DSP, and that canqualitatively increase the value of an image.

It is an additional objective of the present invention to provide aninformation terminal having a function that, in accordance, for example,with a use state, flexibly increases image quality one level and exceedsthat of an information terminal equipped with a camera having the samefunction.

It is a further objective of the present invention to provide aninformation terminal that can improve the image quality, either withoutrequiring a change in the hardware architecture of a camera modulemounted on the information terminal or without increasing the size of acamera DSP.

To achieve these objectives, according to a first invention of thepresent invention, an information terminal is characterized bycomprising:

data operation processing means, for performing operation processing forinput image data and preparing output image data;

removal means, for removing, from the output image data, part or all ofthe steps of an operational processing sequence performed for the inputimage data; and

data processing means, for performing other operational processing stepsfor data obtained by the removal means and for preparing output imagedata.

The image data input to the information terminal are, specifically,color image data based on a Bayer-type RGB arrangement, i.e., anarrangement of red, blue and green pixels that generally is used by asingle-chip image sensor of a color camera.

The data processing means for the information terminal is specific meansfor performing interpolation processing, and more specifically, ismeans, when a sensor outputs R, G or B pixels, for employing the valuesof adjacent pixels having the same color to perform interpolationprocessing using the nearest neighbor method, the linear approximationmethod or the cubic convolution method for pixels G and B, R and B, or Rand G that have been eliminated by a color filter that is laid on thesurface of the sensor.

It is preferable, when irreversible processing has been is performed forthe image data input to the information terminal and the output imagedata have been prepared, that the removal means perform inverse functiontransformation processing to remove part or all of the interpolationprocessing performed for the input data.

It is preferable that the data processing means of the informationterminal again perform an edge enhancement process.

Furthermore, according to a second invention of the present invention,an information terminal is characterized by comprising:

a camera module, which has an image sensor for digitizing and using asignal output by the image sensor to prepare first Bayer-type data, andfor employing the first Bayer-type data to prepare first image data,using a first algorithm, and transmitting the first image data; and

a host module, having a main storage device, for receiving the firstimage data obtained by the camera module and storing the received firstimage data in the main storage device, for reading the first image datafrom the main storage device and extracting second Bayer-type data fromthe first image data that have been read, and for employing the secondBayer-type data to prepare second image data using a second algorithm.

It is preferable that the first image data and/or the second image datahave an RGB form or a YUV form.

It is preferable that the first image data provided be compressed.

It is preferable that the host module include a data output unit foroutputting data to a printer, so that the second image data are outputto the printer through the data output unit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram showing the periphery of a retrialmodule according to one mode of the present invention;

FIG. 2 is a functional block diagram showing the periphery of theretrial module according to the mode of the present invention;

FIG. 3 is a functional block diagram showing the periphery of theretrial module according to the mode of the present invention;

FIG. 4 is a flowchart for the retrial module according to the mode ofthe present invention;

FIG. 5 is a diagram showing a relationship between the purposes of theuse of an information terminal and requested items;

FIG. 6 is a diagram for explaining a first method for the interpolationprocessing;

FIG. 7 is a diagram for explaining a second method for the interpolationprocessing;

FIG. 8 is a diagram for explaining a third method for the interpolationprocessing;

FIG. 9 is a diagram showing the external appearance of a cellular phoneequipped with a camera;

FIG. 10 is a diagram showing the hardware configuration of the cellularphone equipped with the camera;

FIG. 11 is a diagram for explaining the creation and re-creation ofimage data; and

FIG. 12 is a flowchart for a photographing mode.

BEST MODES FOR CARRYING OUT THE INVENTION

1. First Mode

A first mode of the present invention will now be described whilereferring to the accompanying drawings.

(1) Explanation for Encoding Photographed Images

FIG. 1 is a functional block diagram for explaining this mode of theinvention, and for showing only modules of an information terminal thatare required to explain the present invention. In this mode, aphotographed image is encoded into a predetermined form (a JPEG form inFIG. 1) to be stored or to be transmitted, for example.

A camera module 103 includes a sensor and a camera DSP. The sensor has,for example, a color filter to cope with the three primary colors oflight, blue (B), green (G) and red (R), and in this filter, G isarranged in checkerboardwise and R and B are sequentially arrangedlinearly. In this specification, this arrangement is called a Bayer-typearrangement. Image data that are output by the camera module thatemploys the color filter having the Bayer-type arrangement also have aBayer-type arrangement.

When a user points a lens 101 at a subject and releases the shutter, thesensor generates image data that include a Bayer-type arrangementconsonant with the detected subject, and the camera DSP performsoperations that are image quality correction means that includes gammacorrection means, color interpolation means and an edge enhancementprocess and other processes.

During these operations, the individual color elements of the image datahaving the Bayer-type arrangement, i.e., the R element, the G elementand the B element, are extracted; the color correction means adjusts thebalance of the color elements; the gamma correction means transforms thevalues of the pixels for the individual color elements independently, byusing transformation expressions or a lookup table; and the colorinterpolation means performs interpolation processing, so that missingcolor elements, for the pixel positions of the individual colorelements, are provided, for adjacent pixels having the same colorelements. As a result, the color elements that are removed by the colorfilter laid on the image sensor are restored for all the pixels.

In this case, since minimization of the chip area and reductions in sizeand cost are requested, problems could occur in the operating ability ofthe camera DSP; for example, a complicated operation may be disabledthat requires many operations or a large capacity processing memory,such as a line memory, to perform image processing, an extended periodof time may be required even when such a complicated operation isenabled, or synchronization with a succeeding processing step ofreceiving processing results may not be obtained.

Therefore, the color correction means, the gamma correction means andthe color interpolation means available at this stage are those forwhich the operational processing and the processing memory size arereduced as much as possible. In particular, for the color interpolationmeans, which is one of the image processing means that most requiresmemory and operational processing resources, an example can be providedthat, relative to adjacent pixels, for example, employs a linearinterpolation method.

Similarly, for the edge enhancement processing that, comparatively,consumes as many processing resources, there are many examples wherein,in order to optimize the amount of processing memory and the amount ofoperational processing, edge enhancement filters having horizontallyelongated and vertically shortened taps are employed that can performthe processing but that use a smaller number of line memories.

Several other image quality correction processes are performed for theimage data obtained by the above described various processing means, andafter the processing sequence for the camera is completed and when thestorage of image data is the purpose, compression processing, such asJPEG compression, is performed for the image data, and the compressedimage data (hereinafter referred to as processed data) are output tostorage means 111. When the display of image data is the purpose, imagequality correction processing is performed for the image data to obtainthe optimally limited number of pixels and number of colors for displaymeans 113, and the thus obtained image data are output to the displaymeans 113.

A retrial module 107 includes removal means 107 a, color interpolationmeans 107 b and image quality correction means 107 c, and data output bythe camera module, i.e., processed data, are transmitted to the removalmeans 107 a.

When the retrial module 107 is started, the removal means 107 a removespart or all of the image processing that has been performed for theprocessed data by the camera DSP. The processes removed here are, forexample, the color interpolation process, performed for image datahaving a Bayer-type arrangement, and/or the edge enhancement process,which is a typical step performed as part of another image qualityimprovement process.

When the interpolation process has been removed from the interpolateddata, data having a Bayer-type arrangement are recreated. In this case,the removal of the interpolation process consists of the performance ofan intermittent process by which the R, G and B elements of a colorimage, obtained by the series of camera processing events, are arrangedin consonance with the R, G and B Bayer-type arrangement for the imagesensor.

When the color elements of the individual pixels obtained through theperformance of the color interpolation process are reset, restrictionsimposed on the color interpolation processing and the image qualityimprovement processing that are provided by the camera DSP are reducedto their lowest levels.

Thereafter, the data obtained in the removal process are transmitted tothe image quality correction means 107 c, the retrial module 107performs an interpolation process that differs from that performed bythe camera DSP and, if needed, an edge enhancement process that differsfrom that included in the image quality improvement processing for thecamera DSP, and the resultant data are transmitted to JPEG encodingmeans 109.

An interpolation process that requires a more complicated operation, anincreased amount of operational processing and a greater amount ofprocessing line memory can be employed as another interpolation method.Similarly, a process that requires a more complicated operation, anincreased amount of operational processing and a greater amount ofprocessing memory can be employed as an edge enhancement process that isincluded in other image quality improvement processing.

According to this mode, for example, an interpolation process that usesthe nearest neighbor method is provided by the camera DSP, and isperformed by employing an amount of processing memory that is theequivalent of two lines of image data, without any multiplication beingrequired. Then, the retrial module 107 employs an interpolation process,using a cubic convolution method, to perform floating-pointmultiplication for a quantity of processing memory that is theequivalent of five lines of image data.

Furthermore, the interpolation process performed by the camera DSPincludes the image quality improvement process that uses an edgeenhancement filter wherein taps for five horizontally arranged pixelsare provided for each line of an image. The retrial module 107 providesthe image quality improving process, using edge enhancement filters, inwhich taps for five horizontal pixels are provided for each three linegroup in an image, that are changed in accordance with the set of valuesin each block of 3×5 pixels.

Of course, the present invention is not limited to these specific colorinterpolation and edge enhancement processes, and has a generalprocessing architecture.

According to this mode, only when the retrial of the interpolationprocess is required is the retrial module 107 started, and when theretrial is not required, processed data are transmitted directly fromthe camera module 103 to the JPEG encoding means 109.

Whether or not the retrial is required is visually determined when animage is photographed and is displayed on the screen provided by theinformation terminal.

The start of the retrial module 107 is instructed, by the manipulationof a user menu or a key button prepared for the information terminal,when it is visually determined that the image quality is unsatisfactory,such as when an image is slightly blurred or there are jaggies along theedges.

Of course, processing means for automatically determining whether animage quality is satisfactory, and processing means that employ theresults obtained by automatic determination to automatically start theretrial module 107 may be provided.

The retrial module 107 may employ processing that differs from theprocessing provided by the camera DSP, in addition to such image qualityimprovement processing as the interpolation process and the edgeenhancement process, or may employ different processes as only parts ofthese processes.

In the mode described above, before the JPEG compression process isperformed, immediately after an image is photographed, whether theretrial module 107 should be started is determined and the processingincluded for the retrial module is performed. However, as is shown inFIG. 3, these process sequences may be performed for an image obtainedusing JPEG compression.

That is, JPEG decoding means 305 performs the JPEG decoding processingfor JPEG compressed image data and displays the resultant data on ascreen, and a retrial module 307 is started and the processing resultsare transmitted as processed data to JPEG encoding means 309.

(2) Explanation for Outputting of Decoded Image

FIG. 2 is a functional block diagram for explaining another modeaccording to the present invention, and shows only modules thatconstitute an information terminal according to the present inventionand that are required to explain this invention. According to this mode,an image obtained as a predetermined form (a JPEG form in FIG. 2) isoutput and stored.

An explanation will be given for an example wherein JPEG decoding means205 decodes a JPEG image, again performs the processing for the obtainedimage, and outputs the results.

A retrial module 207 includes removal means 207 a, color interpolationmeans 207 b and image quality correction means 207 c, and the JPEGdecoding means 205 transmits JPEG decoded image data to the removalmeans 207 a.

When the retrial module 207 is started, the removal means 207 a removespart or all of the image quality improvement processing, including aninterpolation process (nearest neighbor interpolation, as a specificexample) and an edge enhancement process, that has been performed forthe decoded data.

Thereafter, the data obtained in the removal process are transmitted tothe color interpolation means 207 b, which then performs anotherinterpolation process and another edge enhancement process (fixed edgeenhancement process with five taps for one line), and transmits theobtained data to display means 213.

According to this mode, an example wherein the linear approximationmethod or the cubic convolution method is used can be employed asanother interpolation method. Further, an example wherein a flexibleedge enhancement filter is used, for local characteristics of an imagewherein five taps are provided for three lines, can also be employed.

According to this mode, when a retrial of the interpolation process isrequired, the retrial module 207 is started, and when a retrial is notrequired, the JPEG encoding means 205 transmits data, unchanged, to thedisplay means 213. The retrial module 207 may be started when a retrialof the edge enhancement process is required, and may perform theinterpolation process and the edge enhancement process and then performthe processing in the above described manner.

(3) Processing Performed by Retrial Module

FIG. 4 is a flowchart for explaining the processing performed by theretrial module.

When the retrial module is started (S401), first, the module removespart or all of the image quality improvement processing, including theinterpolation process and the edge enhancement process, that have beenperformed for input data (S403).

Generally, the sequence for the performance of these processes consistsof a plurality of steps, and whether all the processes performed at allthe steps should be removed or only a process performed at apredetermined step should be removed can be determined in accordancewith the situation.

The processing performed for the input data includes a substantiallyreversible process and a substantially irreversible process. Whether asubstantially irreversible process (e.g., the color elements of pixelsfor an image that is obtained by performing the interpolation processfor Bayer-arranged data using the nearest neighbor method) is performeddepends on whether a substantially reversible process, i.e., for thevalues of the original individual pixels that were present prior to theinterpolation process, was performed for the Bayer-arranged data.

The pixels obtained by interpolation and the pixels used for theinterpolation process, i.e., the color elements of the pixels thatcorrespond to the Bayer arrangement, are classified, and informationrelating to the original Bayer-arranged data can be recreated based onthe image data. As a result, the correction process can be easilyremoved.

Actually, there are several variations of the Bayer arrangement.Therefore, in order to identify which color elements of which pixelswere pixels that correspond to the Bayer arrangement, or which pixelswere obtained by interpolation, the pattern of the Bayer arrangement forthe image sensor of the camera must be stored in the informationterminal, and must be referred to when the retrial process is to beperformed.

The same function can be provided in such a manner that the cameramodule outputs the pattern for a Bayer arrangement as a cameraprocessing parameter that is accompanied by image data, and a dataparser is provided for the information terminal so as to separate imagedata from the parameter information and extract the pattern for theBayer arrangement.

Further, to preferably perform the same processing for image data thatare obtained and transmitted by another information terminal, or animage that was previously photographed and has been stored, a fileformat and an additional information file are prepared for the storage,with the image data, of the pattern for a Bayer arrangement.

It should be noted, however, that the image quality can be improved bypresuming a default Bayer arrangement, instead of obtaining a correctBayer arrangement that was employed, and by preforming the retrialprocess as described above.

When the correction/removal process (S403) is completed, a newlyemployed method for a correction process is selected (S405). The methodselection may be manually performed by a user, or may be automaticallyperformed by the terminal in accordance with the use state and variousconditions.

In a case wherein, for example, an interpolation process, such as aprocess performed using the nearest neighbor method, that does notrequire a large amount of operational processing and a great amount ofprocessing memory has been performed for the original image data, aprocess by which a natural image is obtained, e.g., a process using thelinear approximation method or the cubic convolution method, isperformed, for example, to improve the image quality. In this manner,the re-interpolation process can be performed (S409 or S411). On theother hand, when the processing speed has a higher priority than has theimage quality, the re-interpolation process can be performed by usingthe nearest neighbor method (S407). Further, when edge enhancement isrequired, a corresponding process is performed (S415) and the processingis terminated.

(4) Explanation for Interpolation Method Employed in this Mode

The nearest neighbor method, the linear approximation method and thecubic convolution method will now be described while referring to FIGS.6 to 8. As is shown in FIG. 6, the nearest neighbor method is a methodwhereby, when pixel P0 is to be generated between pixels P2 and P3 ofimage data consisting of pixels P1, P2, P3 and P4, the pixel value ofthe nearest pixel P2 is copied to the pixel value of the pixel P0 (seeFIG. 6( b)). Therefore, the amount of operational processing is zero,and merely one line is sufficient for the processing line memoryrequired for the interpolation process.

On the other hand, as is shown in FIG. 8, the linear approximationmethod is a method performed while assuming that the pixel values arelinearly changed, and whereby the pixel value of interpolation target P0is calculated by proportionally distributing the pixel values ofpreceding and succeeding pixels P2 and P3 in accordance with thedistance between the two. That is, when the pixel pitch is normalizedand the distance to the preceding pixel P2 is defined as X, the pixelvalue of the pixel P0 is represented by weighted addition using theneighboring pixels P2 and P3, which is shown in a formula below. In thisexplanation, the process has been performed in a one-dimensionaldirection; however, in actuality, the process performed for the image isperformed horizontally and vertically. Since the pixels corresponding tothe two pixels P2 and P3 are required in both the horizontal directionand the vertical direction to obtain the pixel P0, a processing memoryconsisting of a minimum of two image lines is required. In the followingformulas, the pixel values are denoted by symbols for correspondingpixels to make them easy to understand.

$\begin{matrix}{{{\Delta\; p} = {X\left( {{P\; 3} - {P\; 2}} \right)}}\begin{matrix}{{P\; 0} = {{\Delta\; p} + {P\; 2}}} \\{= {{X\left( {{P\; 3} - {P\; 2}} \right)} + {P\; 2}}} \\{= {{\left( {1 - X} \right)P\; 2} + {{XP}\; 3}}}\end{matrix}} & (1)\end{matrix}$

Whereas, according to the cubic convolution method, the changes in thepixel values are processed to obtain an approximate cubic curve. Thatis, as is shown in FIG. 7, according to the cubic convolution method,the pixel value of pixel P0 is represented by weighted addition usingthe four neighboring pixels P1 to P4, as is shown in the formula below.For this explanation, the process has been performed in aone-dimensional direction; however, in actuality, the process performedfor an image is performed horizontally and vertically. Since pixelscorresponding to the four pixels P1, P2, P3 and P4 are required bothhorizontally and vertically to obtain the pixel P0, a processing memoryconsisting of a minimum of four image lines is required.P0=k1·P1+k2·P2+k3·P3+k4·P4  (2)

In this formula, k1 to k4 denote weighted coefficients, and are obtainedby the following interpolation function h(t). In this formula, t denotesa variable t1 to t4 that represents the positional relationship betweeneach of the pixels P1 to P4 and the processing target pixel P0, and isrepresented by using the above described distance X.

$\begin{matrix}{{{{\left( {a + 2} \right){t}^{3}} - {\left( {a + 3} \right){t}^{2}} + 10} \leq {t} < 1}\mspace{14mu}\begin{matrix}{{h(t)} = {{a{t}^{3}} - {5a{t}^{2}} + {8a{t}} - {4a}}} & {1 \leq {t} < 2} \\0 & {2 \leq {t}}\end{matrix}{{wherein}\mspace{14mu}\begin{matrix}{{t\; 1} = {1.0 + X}} \\{{t\; 2} = {+ X}} \\{{t\; 3} = {1.0 - X}} \\{{t\; 4} = {2.0 - X}}\end{matrix}}} & (3)\end{matrix}$

Since the nearest neighbor method is a process merely for determining adistance from the preceding pixel to the succeeding pixel and forsetting a pixel value, a drawback exists in that the image quality isdegraded while the processing speed is extremely short.

On the other hand, according to formula (1), the linear approximationmethod requires two multiplications and one addition, so that comparedwith the nearest neighbor method, the amount of operational processingis increased, the processing speed is reduced, and a minimum two linesof processing line memory and a greater amount of memory are required.However, as a characteristic, the image quality is superior to thatobtained using the nearest neighbor method.

Furthermore, the cubic convolution method requires four multiplicationsand three additions even for only the weighted addition, so that ofthese three methods, this method requires the largest number ofoperations, and accordingly, the processing speed is the lowest.However, as a characteristic, the highest image quality is provided.

Therefore, in the method selection process performed by the retrialmodule (S405 in FIG. 4), the optimal method need only be selected fromamong the above described methods, so that an image quality and aprocessing speed can be provided in consonance with the use state andthe purpose of the use.

2. Second Mode

A second mode of the present invention will now be described whilereferring to the accompanying drawings.

FIG. 9 is a diagram showing the external appearance of a cellular phoneequipped with a camera, for which the present invention is applied. Adisplay panel 2, a ten-key pad 3, a function button 4, an antenna 5,etc., are provided on the obverse surface of a camera-equipped cellularphone 1; a camera unit 11, a battery cover 9, etc., are provided on thereverse surface; and a case 6, which holds all these components, isprovided. Further, a data output terminal 41 is provided for theconnection of an external printer. As is well known, a camera-equippedcellular phone 1 is small and light, so that no inconvenience isincurred when the cellular phone is carried in a handbag. The cameraunit 11 includes a lens 7 and an LED luminaire 8, and is constituted asa camera module having an independent case 10. The camera unit isprovided as an independent module because multiple uses for the cameramodule are available since the camera module can be easily assembledwith another cellular phone or a PDA. Therefore, the camera-equippedcellular phone 1 of this mode can be separated into the camera module 11and the remaining section (the host module).

The function button 4, for example, is used to send or receive a call,or is used as a shutter button for photographing. The function button 4is also used as a button to start the reconstruction of image data. Whena user employs the camera-equipped cellular phone 1 to make a call, theuser holds the camera-equipped cellular phone 1 with the lens 7 aimed ata target, and confirms, on the display panel 2, a preview image obtainedby the camera module 11. Then, imaging is performed by pressing thefunction button 4, and the obtained image data are stored in the storagedevice provided for the camera-equipped cellular phone 1.

While referring to FIG. 10, an explanation will be given for thehardware configuration and the operation of the camera-equipped cellularphone for which the present invention is applied. As is described above,the camera-equipped cellular phone 1 is constituted by the camera module11 and a host module 12; the camera module 11 controls the imaging andthe creation of image data, and the host module 12 controls the savingand the display of the created image data and PDA functions, such as aphone function and a scheduling function.

The camera module 11 includes the lens 7, the LED luminaire 8, a solidstate imaging device 13, an A/D converter 14, an image data constructionunit 16, a JPEG compression unit 17, a bus 18, a data interface 19 and acontrol interface 20. Of these components, the image data constructionunit 16, the JPEG compression unit 17, the bus 18, the data interface 19and the control interface 20 are provided as a camera LSI 15 that uses asingle chip.

A CCD or a CMOS sensor, for example, can be employed as the solid stateimaging device). The solid-state imaging device 13 performs imaging byconverting, into an electrical signal, light that has passed through thelens 7. The signal output by the solid state imaging device 13 isconverted into digital data by the A/D converter 14. The digital dataare called Bayer-type data, and are not yet image data that can bedisplayed by a computer or that can be printed by a printer.

Image data are constructed by the image data construction unit 16. Theimage data construction unit 16 first performs the original imageprocessing, such as lens density correction and white balance control,for the Bayer-type data. Then, the Bayer-type data for which theoriginal image processing has been performed are separated into redelements (R), green elements (G) and blue elements (B), and the CFA(Color Filter Array) interpolation process is performed for these colorelements to construct RGB image data consisting of three RGB planes.

The algorithm used for the CFA interpolation very much affects thequality of constructed image data. It is difficult for image data havinga high quality to be obtained using an algorithm available for a cameraLSI that limits size, cost and speed. The image data construction unit16 performs the processing, such as edge enhancement or gammacorrection, for the thus obtained image data.

Finally, the RGB form of the image data is changed to the YUV form. Theimage data constituting one frame are sequentially produced for eachline or for every several lines, and in the end, the image data for thewhole frame are constructed by employing signals that are output by thesolid state imaging device 13 using single image photography. Each setof the thus produced image data is sequentially transmitted to the JPEGcompression unit 17, which then performs JPEG compression for the imagedata and transmits the resultant data, through the data interface 19, tothe host module 12.

Referring again to FIG. 10, the hardware configuration and the operationof the host module 12 will be described. The host module 12 includes adata interface 25, a control interface 26, a CPU 30, a bus 24, aninterface 31 for a temporary storage device, a temporary storage device32, an interface 33 for a main storage device, a main storage device 34,an interface 35 for a display device, a display device 36, an interface37 for a keyboard, a keyboard 38, an interface 40 for a printer, abaseband controller 22, and an antenna 23.

Of these components, the CPU 30, the bus 24 and the interfaces 25, 26,31, 33, 35, 37 and 40 are provided as an application engine 21 by usinga single chip. The baseband controller 22 controls a function related tothe sending or the reception of calls, and has a special CPU. Theapplication engine 21 controls a function other than the calltransmission or reception function, and also controls functions for theprocessing of image data received from the camera module 11, forcontrolling the keyboard 38, for playing games, for playing music, andfor scheduling. The CPU 30 can provide, through the control interfaces26 and 20 and the bus 18, not only control of the JPEG compression unit17 of the camera module 11, but also control of the individual processesperformed by the image data construction unit 16, the turning on or offof the LED luminaire 8, the changing of the data collection mode of thesolid state imaging device, and parameter control for the A/D converter14. The keyboard 38 includes a ten-key pad 3 and a function button 4.

The image data output by the camera module 11 are transmitted throughthe data interface 25 to the host module 12, and are temporarily storedin the temporary storage device 32. A typical temporary storage device32 is an SDRAM. The CPU 30 thereafter reads image data from thetemporary storage device 32, and stores the image data in the mainstorage device 34. The main storage device 34 includes a storage mediumon which data can be continuously saved when the camera-equippedcellular phone 1 has been powered off, and can, for example, be a flashmemory, a CF card or an SD card.

The camera-equipped cellular phone of the present invention canreconstruct image data by using the host module. The CPU 30 performs thereconstruction of image data upon the reception of an instruction from auser interface. The CPU 30 first processes the image data to restore theBayer-type data, and then employs a high-level algorithm to constructnew image data. The thus constructed high quality image data are thenstored in the main storage device 34, or are transmitted to a printerthrough the printer interface 40.

The processing for the construction and reconstruction of image datawill be described while referring to FIG. 11.

Data obtained merely by digitizing the output signal of the sensor, asis indicated by number 1144, is of Bayer type, wherein multiple unitmatrixes, each of which is composed of one red pixel (R), two greenpixels (G) and one glue pixel (B), are repeated. The elements of theindividual colors are separated from the Bayer-type data, and aninterpolation process is performed for these elements to construct RGBimage data 45. Pixel data enclosed by circles, as denoted by number1145, are either the same as the original Bayer-type data or aresubstantially unchanged. Therefore, the pixel data enclosed by circles,denoted by 1145, need only be extracted to reconstruct the originalBayer-type data.

Of course, in order to reconstruct the original Bayer-type data, whichpixel data of the RGB image data are derived from the originalBayer-type data must be determined in advance. There are various methodsavailable for this determination: a method for performing thisdetermination at the interpolation stage; and a method for embeddinginformation in the header of the constructed image data. When thehigh-level original image processing, the interpolation process and thepost-process are performed for the reconstructed Bayer-type data, newimage data 1147 are created.

It should be noted that when the Bayer-type data are reconstructed fromthe JPEG compressed RGB data, Bayer-type data that are completely thesame as the original data can not be extracted. However, the advantageobtained when the image data are reconstructed by using a high-levelalgorithm is greater than the loss of some data.

The processing performed by the camera-equipped cellular phone 1 of thepresent invention will be now described while referring to FIG. 12.

In the case of photography, first, the user interface is manipulated(S1201). Then, the image sensor of the camera module 11 converts lightinto an electrical signal (S1202), and image data are constructed basedon the electrical signal (S1203). The thus constructed image data arecompressed using the JPEG method (S1204), and the compressed data aretransmitted to the host module 12 (S1205). The host module receives theimage data (S1206), and stores the image data in the storage device(S1207).

In the case of the printing of obtained image data, first, the userinterface is manipulated (S1208). Then, the host module 12 reads imagedata from the storage device (S1209), decompresses the image data thathave been read (S1210), and extracts and reconstructs Bayer-type databased on the decompressed image data (S1211). Further, the host module12 employs a new high-level algorithm to prepare new image data based onthe thus reconstructed Bayer-type data (step S1212), and outputs thenewly created image data to the printer (step S1213).

The modes of the present invention have been explained; however, thepresent invention is not limited to the above description. While thenearest neighbor method, the linear approximation method or the cubicconvolution method has been employed in these modes as the interpolationmethod, an interpolation process using, for example, the spline functionor the Bezier function can be employed instead of, or in addition to theabove methods.

Furthermore, in the modes, the present invention has been applied forthe image data processing. However, the present invention is not limitedto this, and can be applied for a case wherein there is a change in thesampling pitch of various data, e.g., the sampling frequency of audiodata.

INDUSTRIAL APPLICABILITY

As is described above, according to the present invention, aninformation terminal can be provided that can change, at a low cost andwithin a short period of time, image quality and processing speed inaccordance with the use state and a variety of conditions.

Furthermore, according to the present invention, an information terminalcan be provided that can reduce the affect produced by the simplifiedprocessing performed by the camera DSP, and that can improve the imagequality of an image.

Further, according to the present invention, an information terminal canbe provided that can produce an image having a higher image quality thanother information terminals that employ cameras having the samefunctions.

In addition, according to the present invention, an information terminalcan be provided that can improve the quality of an image withoutchanging a camera module or increasing the functions required of thecamera DSP.

1. An apparatus comprising: at least one processor; and at least onememory including non-transitory computer readable program code, wherethe at least one memory and the computer program code are configured,with the at least one processor, to cause the apparatus to at least:perform at least one process for input image data resulting from a firstinterpolation process, including removing at least part of an effect ofthe first interpolation process from the input image data; and prepareoutput data by performing a second, different, interpolation process forthe image data obtained after the at least part of the effect of thefirst interpolation has been removed.
 2. The apparatus according toclaim 1, comprising a camera module including a lens, an image sensor,and a camera digital signal processor, where the camera digital signalprocessor includes a color correction unit, a gamma correction unit, acolor interpolation unit, and an image quality correction unit, wherethe input image data is produced by the camera module, where removingthe at least part of the effect of the first interpolation processincludes removing pixels that are interpolated by the color correctionunit of the camera digital signal processor, and minimizing affects dueto a color correction process and an image quality correction processperformed by the camera digital signal processor, and where performingthe second, different, interpolation process includes performing a colorinterpolation process that is different than the first interpolationprocess and further performing an image quality correction process. 3.The apparatus according to claim 2, where the apparatus is furthercaused to: identify an arrangement pattern for color filters that arelaid on the image sensor, separate color elements of pixels generatedduring the interpolation process by the color correction unit from colorelements of pixels used to produce the color elements of pixels, andselectively perform a process for the color elements of the pixelsgenerated during the interpolation process by the color correction unit.4. The apparatus of claim 1 embodied in a mobile device.
 5. A method,comprising: obtaining image data, performing an interpolation processesfor the image data thus obtained, and outputting the image dataresulting from the interpolation processes, where performing theinterpolation processes include: performing at least one processincluding removing, with at least one processor, at least part of aneffect of a first interpolation process from the image data resultingfrom the first interpolation process; and performing a second,different, interpolation process for image data obtained after the atleast part of the effect of the first interpolation process has beenremoved.
 6. The method of claim 5, wherein the at least one processincludes a process performed on pixels that are interpolated by a colorcorrector so as to minimize affects due to a color correction processand an image quality correction process performed by a camera digitalsignal processor, and where the second, different, interpolation processincludes a color interpolation process and an image quality correctionprocess that is different than the first interpolation process.
 7. Themethod of claim 6, where performing the at least one process furtherincludes: identifying an arrangement pattern for color filters that arelaid on the image sensor, separating color elements of pixels generatedduring the first interpolation processes from color elements of pixelsused to produce the color elements of pixels, and selectively processingcolor elements of pixels generated during the first interpolationprocess.